The False Negative Rate of Confound Checks
(A2021-92617)

AUTHORS

Niels van de Ven, Tilburg University

ABSTRACT

Researcher degrees of freedom tend to inflate the chance of false positive effects: researchers who want to find an effect are more likely to find one even when none exists (Simmons et al., 2011). However, for confound checks (that aim to rule out alternative explanations) a researcher does not want to find an effect, so researcher degrees of freedom likely increase the false negative rate. We present a case study on how mood is measured (425 articles with 827 studies from JCR, JMR, & JPSP). Measurement practices are worse when mood is added as a confound check than when used as a manipulation check or dependent variable (e.g., reliability is less frequently reported, sensitivity information is more often missing). The most frequent mood measure used as a confound check is the PANAS, and an experiment (N = 282) finds that this measure is less sensitive than another mood measure, and its measurement properties are very poor (with extremely high measurement error). Bad measurement practices increase the false negative rate and we find them to be more likely for confound checks. We conclude that especially for confound checks measurement validity should be taken more seriously.